File system image transfer

ABSTRACT

The invention provides a method and system for duplicating all or part of a file system while maintaining consistent copies of the file system. The file server maintains a set of snapshots, each indicating a set of storage blocks making up a consistent copy of the file system as it was at a known time. Each snapshot can be used for a purpose other than maintaining the coherency of the file system, such as duplicating or transferring a backup copy of the file system to a destination storage medium. In a preferred embodiment, the snapshots can be manipulated to identify sets of storage blocks in the file system for incremental backup or copying, or to provide a file system backup that is both complete and relatively inexpensive.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to storage systems.

2. Related Art

In computer file systems for storing and retrieving information, it issometimes advantageous to duplicate all or part of the file system. Forexample, one purpose for duplicating a file system is to maintain abackup copy of the file system to protect against lost information.Another purpose for duplicating a file system is to provide replicas ofthe data in that file system available at multiple servers, to be ableto share load incurred in accessing that data.

One problem in the known art is that known techniques for duplicatingdata in a file system either are relatively awkward and slow (such asduplication to tape), or are relatively expensive (such as duplicationto an additional set of disk drives). For example, known techniques forduplication to tape rely on logical operations of the file system andthe logical format of the file system. Being relatively cumbersome andslow discourages frequent use, resulting in backup copies that arerelatively stale. When data is lost, the most recent backup copy mightthen be a day old, or several days old, severely reducing the value ofthe backup copy.

Similarly, known techniques for duplication to an additional set of diskdrives rely on the physical format of the file system as stored on theoriginal set of disk drives. These known techniques use an additionalset of disk drives for duplication of the entire file system. Beingrelatively expensive discourages use, particularly for large filesystems. Also, relying on the physical format of the file systemcomplicates operations for restoring backup data and for performingincremental backup.

Accordingly, it would be desirable to provide a method and system forduplicating all or part of a file system, which can operate with anytype of storage medium without either relative complexity or expense,and which can provide all the known functions for data backup andrestore. This advantage is achieved in an embodiment of the invention inwhich consistent copies of the file system are maintained, so thoseconsistent snapshots can be transferred at a storage block level usingthe file server's own block level operations.

SUMMARY OF THE INVENTION

The invention provides a method and system for duplicating all or partof a file system while maintaining consistent copies of the file system.The file server maintains a set of snapshots, each indicating a set ofstorage blocks making up a consistent copy of the file system as it wasat a known time. Each snapshot can be used for a purpose other thanmaintaining the coherency of the file system, such as duplicating ortransferring a backup copy of the file system to a destination storagemedium. In a preferred embodiment, the snapshots can be manipulated toidentify sets of storage blocks in the file system for incrementalbackup or copying, or to provide a file system backup that is bothcomplete and relatively inexpensive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a first system for file system imagetransfer.

FIG. 2 shows a block diagram of a set of snapshots in a system for filesystem image transfer.

FIG. 3 shows a process flow diagram of a method for file system imagetransfer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description, a preferred embodiment of the invention isdescribed with regard to preferred process steps and data structures.However, those skilled in the art would recognize, after perusal of thisapplication, that embodiments of the invention may be implemented usingone or more general purpose processors (or special purpose processorsadapted to the particular process steps and data structures) operatingunder program control, and that implementation of the preferred processsteps and data structures described herein using such equipment wouldnot require undue experimentation or further invention.

Inventions described herein can be used in conjunction with inventionsdescribed in the following applications:

application Ser. No. 08/471,218, filed Jun. 5, 1995, in the name ofinventors David Hitz et al., titled “A Method for Providing Parity in aRaid Sub-System Using Non-Volatile Memory”, attorney docket numberNET-004; now U.S. Pat. No. 5,948,110;

application Ser. No. 08/454,921, filed May 31, 1995, in the name ofinventors David Hitz et al., titled “Write Anywhere File-System Layout”,attorney docket number NET-005; now U.S. Pat. No. 5,819,292;

application Ser. No. 08/464,591, filed May 31, 1995, in the name ofinventors David Hitz et al., titled “Method for Allocating Files in aFile System Integrated with a Raid Disk Sub-System”, attorney docketnumber NET-006 now U.S. Pat. No. 6,038,570.

Each of these applications is hereby incorporated by reference as iffully set forth herein. They are collectively referred to as the “WAFLDisclosures.”

File Servers and File System Image Transfer

FIG. 1 shows a block diagram of a system for file system image transfer.

A system 100 for file system image transfer includes a file server 110and a destination file system 120.

The file server 110 includes a processor 111, a set of program and datamemory 112, and mass storage 113, and preferably is a file server 110like one described in the WAFL Disclosures. In a preferred embodiment,the mass storage 113 includes a RAID storage subsystem.

The destination file system 120 includes mass storage, such as a flashmemory, a magnetic or optical disk drive, a tape drive, or other storagedevice. In a preferred embodiment, the destination file system 120includes a RAID storage subsystem. The destination file system 120 canbe coupled directly or indirectly to the file server 110 using acommunication path 130.

In a first preferred embodiment, the destination file system 120 iscoupled to the file server 110 and controlled by the processor 111similarly to the mass storage 113. In this first preferred embodiment,the communication path 130 includes an internal bus for the file server110, such as an I/O bus, a mezzanine bus, or other system bus.

In a second preferred embodiment, the destination file system 120 isincluded in a second file server 140. The second file server 140,similar to the first file server 110, includes a processor, a set ofprogram and data memory, and mass storage that serves as the destinationfile system 120 with regard to the first file server 110. The secondfile server preferably is a file server like one described in the WAFLDisclosures. In this second preferred embodiment, the communication path130 includes a network path between the first file server 110 and thesecond file server 140, such as a direct communication link, a LAN(local area network), a WAN (wide area network), a NUMA network, oranother interconnect.

In a third preferred embodiment, the communication path 130 includes anintermediate storage medium, such as a tape, and the destination filesystem 120 can be either the first file server 110 itself or a secondfile server 140. As shown below, when the file server 110 selects a setof storage blocks for transfer to the destination file system 120, thatset of storage blocks can be transferred by storing them onto theintermediate storage medium. At a later time, retrieving that set ofstorage blocks from the intermediate storage medium completes thetransfer.

It is an aspect of the invention that there are no particularrestrictions on the communication path 130. For example, a first part ofthe communication path 130 can include a relatively high-speed transferlink, while a second part of the communication path 130 can include anintermediate storage medium.

It is a further aspect of the invention that the destination file system120 can be included in the first file server 110, in a second fileserver 140 110, or distributed among a plurality of file servers 110.Transfer of storage blocks from the first file server 110 to thedestination file system 120 is thus completely general, and includes thepossibility of a wide variety of different file system operations:

Storage blocks from the first file server 110 can be dumped to anintermediate storage medium, such as a tape or a second disk drive,retained for a period of time, and then restored to the first fileserver 110. Thus, the first file server 110 can itself be thedestination file system.

Storage blocks from the first file server 110 can be transferred to asecond file server 140, and used at that second file server 140. Thus,the storage blocks can be copied en masse from the first file server 110to the second file server 140 110.

Storage blocks from the first file server 110 can be distributed using aplurality of different communication paths 130, so that some of thestorage blocks are immediately accessible while others are recorded in arelatively slow intermediate storage medium, such as tape.

storage blocks from the first file server 110 can be selected from acomplete file system, transferred using the communication path 130, andthen processed to form a complete file system at the destination filesystem 120.

In alternative embodiments described herein, the second file server 140can have a second destination file system. That second destination filesystem can be included within the second file server 140, or can beincluded within a third file server 110 similar to the first file server110 or the second file server 140 110.

More generally, each n^(th) file server can have a destination filesystem 120, either included within the n^(th) file server, or includedwithin an n+1^(st) file server. The set of file servers can thus form adirected graph, preferably a tree with the first file server 110 as theroot of that tree.

File System Storage Blocks

As described in the WAFL Disclosures, a file system 114 on the fileserver 110 (and in general, on the n^(th) file server 110), includes aset of storage blocks 115, each of which is stored either in the memory112 or on the mass storage 113. The file system 114 includes a currentblock map, which records which storage blocks 115 are part of the filesystem 114 and which storage blocks 115 are free.

As described in the WAFL Disclosures, the file system on the massstorage 113 is at all times consistent. Thus, the storage blocks 115included in the file system at all times comprise a consistent filesystem 114.

As used herein, the term “consistent,” referring to a file system (or tostorage blocks in a file system), means a set of storage blocks for thatfile system that includes all blocks required for the data and filestructure of that file system. Thus, a consistent file system stands onits own and can be used to identify a state of the file system at somepoint in time that is both complete and self-consistent.

As described in the WAFL Disclosures, when changes to the file system114 are committed to the mass storage 113, the block map is altered toshow those storage blocks 115 that are part of the committed file system114. In a preferred embodiment, the file server 110 updates the filesystem frequently, such as about once each 10 seconds.

Snapshots

FIG. 2 shows a block diagram of a set of snapshots in a system for filesystem image transfer.

As used herein, a “snapshot” is a set of storage blocks, the memberstorage blocks forming a consistent file system, disposed using a datastructure that allows for efficient set management. The efficient setmanagement can include time efficiency for set operations (such aslogical sum, logical difference, membership, add member, remove member).For example, the time efficiency can include O(n) time or less for nstorage blocks. The efficient set management can also include spaceefficiency for enumerating the set (such as association with physicallocation on mass storage or inverting the membership function). Thespace efficiency can mean about 4 bytes or less per 4K storage block ofdisk space, a ratio about 1000:1 better than duplicating the storagespace.

As described herein, the data structure for the snapshot is stored inthe file system so there is no need to traverse the file system tree torecover it. In a preferred embodiment, each snapshot is stored as a filesystem object, such as a blockmap. The blockmap includes a bit planehaving one bit for each storage block, other than bits used to identifyif the storage block is in the active file system.

Moreover, when the file system is backed-up, restored, or otherwisecopied or transferred, the blockmap within the file system is as part ofthe same operation itself also backed-up, restored, or otherwise copiedor transferred. Thus, operations on the file system inherently includepreserving snapshots.

Any particular snapshot can be transferred by any communicationtechnique, including

transfer using storage in an intermediate storage medium (such asnonvolatile memory, tape, disk in the same file system, disk in adifferent file system, or disk distributed over several file systems);

transfer using one or more network messages,

transfer using communication within a single file server or set of fileservers (such as for storage to disk in the same file system, to disk ina different file system, or to disk distributed over several filesystems).

A collection 200 of snapshots 210 includes one bit plane for eachsnapshot 210. Each bit plane indicates a set of selected storage blocks115. In the figure, each column indicates one bit plane (that is, onesnapshot 210), and each row indicates one storage block 115 (that is,the history of that storage block 115 being included in or excluded fromsuccessive snapshots 210). At the intersection of each column and eachrow there is a bit 211 indicating whether that particular storage block115 is included in that particular snapshot 210.

Each snapshot 210 comprises a collection of selected storage blocks 115from the file system 114 that formed all or part of the (consistent)file system 114 at some point in time. A snapshot 210 can be createdbased on to the block map at any time by copying the bits from the blockmap indicating which storage blocks 115 are part of the file system 114into the corresponding bits 211 for the snapshot 210.

Differences between the snapshots 210 and the (active) file system 114include the following:

The file system 114 is a consistent file system 114 that is being usedand perhaps modified, while the snapshots 210 represent copies of thefile system 114 that are read-only.

The file system 114 is updated frequently, while the snapshots 210represent copies of the file system 114 that are from the relativelydistant past.

There is only one active file system 114, while there can be (andtypically are) multiple snapshots 210.

At selected times, the file server 110 creates a new bit plane, based onthe block map, to create a new snapshot 210. As described herein,snapshots 210 are used for backup and mirroring of the file system 114,so in preferred embodiments, new snapshots 210 are created at periodictimes, such as once per hour, day, week, month, or as otherwise directedby an operator of the file server 110.

Storage Images and Image Streams

As used herein a “storage image” includes an indicator of a set ofstorage blocks selected in response to one or more snapshots. Thetechnique for selection can include logical operations on sets (such aspairs) of snapshots. In a preferred embodiment, these logical operationscan include logical sum and logical difference.

As used herein, an “image stream” includes a sequence of storage blocksfrom a storage image. A set of associated block locations for thosestorage blocks from the storage image can be identified in the imagestream either explicitly or implicitly. For a first example, the set ofassociated block locations can be identified explicitly by includingvolume block numbers within the image stream. For a second example, theset of associated block locations can be identified implicitly by theorder in which the storage blocks from the storage image are positionedor transferred within the image stream.

The sequence of storage blocks within the image stream can be optimizedfor a file system operation. For example, the sequence of storage blockswithin the image stream can be optimized for a backup or restore filesystem operation.

In a preferred embodiment, the sequence of storage blocks is optimizedso that copying of an image stream and transfer of that image streamfrom one file server to another is optimized. In particular, thesequence of storage blocks is selected so that storage blocks identifiedin the image stream can be, as much as possible, copied in parallel froma plurality of disks in a RAID file storage system, so as to maximizethe transfer bandwidth from the first file server.

A storage image 220 comprises a set of storage blocks 115 to be copiedfrom the file system 114 to the destination file system 120.

The storage blocks 115 in the storage image 220 are selected so thatwhen copied, they can be combined to form a new consistent file system114 on the destination file system 120. In various preferredembodiments, the storage image 220 that is copied can be combined withstorage blocks 115 from other storage images 220 (which were transferredat earlier times).

As shown herein, the file server 110 creates each storage image 220 inresponse to one or more snapshots 210.

An image stream 230 comprises a sequence of storage blocks 115 from astorage image 220. When the storage image 220 is copied from the filesystem 114, the storage blocks 115 are ordered into the image stream 230and tagged with block location information. When the image stream 230 isreceived at the destination file system 120, the storage blocks 115 inthe image stream 230 are copied onto the destination file system 120 inresponse to the block location information.

Image Addition and Subtraction

The system 100 manipulates the bits 211 in a selected set of storageimages 220 to select sets of storage blocks 115, and thus form a newstorage image 220.

For example, the following different types of manipulation are possible:

The system 100 can form a logical sum of two storage images 220 A+B byforming a set of bits 211 each of which is the logical OR (A v B) of thecorresponding bits 211 in the two storage images 220. The logical sum oftwo storage images 220 A+B is the union of those two storage images 220.

The system 100 can form a logical difference of two storage images 220A−B by 18 forming a set of bits 211 each of which is logical “1” only ifthe corresponding bit 211 A is logical “1” and the corresponding bit 211B is logical “0” in the two storage images 220.

The logical sum of two storage images 220 A+B comprises a storage image220 that includes storage blocks 115 in either of the two originalstorage images 220. Using the logical sum, the system 100 can determinenot just a single past state of the file system 114, but also a historyof past states of that file system 114 that were recorded as snapshots210.

The logical difference of two selected storage images 220 A−B comprisesjust those storage blocks that are included in the storage image 220 Abut not in the storage image 220 B. (To preserve integrity ofincremental storage images, the subtrahend storage image 220 B is alwaysa snapshot 210.) A logical difference is useful for determining astorage image 220 having a set of storage blocks forming an incrementalimage, which can be used in combination with full images.

In alternative embodiments, other and further types of manipulation mayalso be useful. For example, it may be useful to determine a logicalintersection of snapshots 210, so as to determine which storage blocks115 were not changed between those snapshots 210.

In further alternative embodiments, the system 100 may also use the bits211 from each snapshot 210 for other purposes, such as to perform otheroperations on the storage blocks 115 represented by those bits 211.

Incremental Storage Images

As used herein, an “incremental storage image” is a logical differencebetween a first storage image and a second storage image.

As used herein, in the logical difference A−B, the storage image 220 Ais called the “top” storage image 220, and the storage image 220 B iscalled the “base” storage image 220.

When the base storage image 220 B comprises a full set F of storageblocks 115 in a consistent file system 114, the logical difference A−Bincludes those incremental changes to the file system 114 between thebase storage image 220 B and the top storage image 220 A.

Each incremental storage image 220 has a top storage image 220 and abase storage image 220. Incremental storage images 220 can be chainedtogether when there is a sequence of storage images 220 C_(i) where abase storage image 220 for each C_(i) is a 18 top storage image 220 fora next C_(i+1).

Examples of Incremental Images

For a first example, the system 100 can make a snapshot 210 each day,and form a level-0 storage image 220 in response to the logical sum ofdaily snapshots 210.

June3.level0=June3+June2+June1

(June3, June2, and June1 are snapshots 220 taken on those respectivedates.)

The June3.level0 storage image 220 includes all storage blocks 115 inthe daily snapshots 210 June3, June2, and June1. Accordingly, theJune3.level0 storage image 220 includes all storage blocks 115 in aconsistent file system 114 (as well as possibly other storage blocks 115that are unnecessary for the consistent file system 114 active at thetime of the June3 snapshot 210).

In the first example, the system 100 can form an (incremental) level-1storage image 220 in response to the logical sum of daily snapshots 210and the logical difference with a single snapshot 210.

June5.level1=June5+June4−June3

(June5, June4 and June3 are snapshots 220 taken on those respectivedates.)

It is not required to subtract the June2 and Junel snapshots 210 whenforming the June5.level1 storage image 220. All storage blocks 115 thatthe June5 snapshot 210 and the June4 snapshot 210 have in common witheither the June2 snapshot 210 or the Junel snapshot 210, they willnecessarily have in common with the June3 snapshot 210. This is becauseany storage block 115 that was part of the file system 114 on June2 orJune1, and is still part of the file system 114 on June5 or June4, musthave also been part of the file system 114 on June3.

In the first example, the system 100 can form an (incremental) level-2storage image 220 in response to the logical sum of daily snapshots 210and the logical difference with a single snapshot 210 from the time ofthe level-1 base storage image 220.

June7.level2=June7+June6−June5

(June7, June6, and June5 are snapshots 210 taken on those respectivedates.)

In the first example, the storage images 220 June3.level0, June5.level1,and June7.level2 collectively include all storage blocks 115 needed toconstruct a full set F of storage blocks 115 in a consistent file system114.

For a second example, the system 100 can form a different (incremental)level-1 storage image 220 in response to the logical sum of dailysnapshots 210 and the logical difference with a single snapshot 210 fromthe time of the level-0 storage image 220.

June9.level1=June9+June8−June3

(June9, June8, and June3 are snapshots 210 taken on those respectivedates.)

Similar to the first example, the storage images 220 June3.level0 andJune9.level1 collectively include all storage blocks 115 needed toconstruct a full set F of storage blocks 115 in a consistent file system114. There is no particular requirement that the June9.level1 storageimage 220 be related to or used in conjunction with the June7.level2storage image 220 in any way.

File System Image Transfer Techniques

To perform one of these copying operations, the file server 110 includesoperating system or application software for controlling the processor111, and data paths for transferring data from the mass storage 113 tothe communication path 130 to the destination file system 120. However,the selected storage blocks 115 in the image stream 230 are copied fromthe file system 114 to the corresponding destination file system 120without logical file system processing by the file system 114 on thefirst file server 110.

In a preferred embodiment, the system 100 is disposed to perform one ofat least four such copying operations:

Volume Copying. The system 100 can be disposed to create an image stream230 for copying the file system 114 to the destination file system 120.

The image stream 230 comprises a sequence of storage blocks 115 from astorage image 220. As in nearly all the image transfer techniquesdescribed herein, that storage image 220 can represent a full image oran incremental image:

Full image: The storage blocks 115 and the storage image 220 represent acomplete and consistent file system 114.

Incremental image: The storage blocks 115 and the storage image 220represent an incremental set of changes to a consistent file system 114,which when combined with that file system 114 form a new consistent filesystem 114.

The image stream 230 can be copied from the file server 110 to thedestination file system 120 using any communication technique. Thiscould include a direct communication link, a LAN (local area network), aWAN (wide area network), transfer via tape, or a combination thereof.When the image stream 230 is transferred using a network, the storageblocks 115 are encapsulated in messages using a network communicationprotocol known to the file server 110 and to the destination file system120. In some network communication protocols, there can be additionalmessages between the file server 110 and to the destination file system120 to ensure the receipt of a complete and correct copy of the imagestream 230.

The destination file system 120 receives the image stream 230 andidentifies the storage blocks 115 from the mass storage 113 to berecorded on the destination file system 120.

When the storage blocks 115 represent a complete and consistent filesystem 114, the destination file system 120 records that file system 114without logical change. The destination file system 120 can make thatfile system 114 available for read-only access by local processes. Inalternative embodiments, the destination file system 120 may make thatfile system 114 available for access by local processes, without makingchanges by those local processes available to the file server 110 thatwas the source of the file system 114.

When the storage blocks 115 represent an incremental set of changes to aconsistent file system 114, the destination file system 120 combinesthose changes with that file system 114 form a new consistent filesystem 114. The destination file system 120 can make that new filesystem 114 available for read-only access by local processes.

In embodiments where the destination file system 120 makes thetransferred file system 114 available for access by local processes,changes to the file system 114 at the destination file system 120 can beflushed when a subsequent incremental set of changes is received by thedestination file system 120.

All aspects of the file system 114 are included in the image stream 230,including file data, file structure hierarchy, and file attributes. Fileattributes preferably include NFS attributes, CIFS attributes, and thosesnapshots 210 already maintained in the file system 114.

Disk Copying. In a first preferred embodiment of volume copying (hereincalled “disk copying”), the destination file system 120 can include adisk drive or other similar accessible storage device. The system 100can copy the storage blocks 115 from the mass storage 113 to thataccessible storage device, providing a copy of the file system 114 thatcan be inspected at the current time.

When performing disk copying, the system 100 creates an image stream230, and copies the selected storage blocks 115 from the mass storage113 at the file server 110 to corresponding locations on the destinationfile system 120. Because the mass storage 113 at the file server 110 andthe destination file system 120 are both disk drives, copying tocorresponding locations should be simple and effective.

It is possible that locations of storage blocks 115 at the mass storage113 at the file server 110 and at the destination file system 120 do notreadily coincide, such as if the mass storage 113 and the destinationfile system 120 have different sizes or formatting. In those cases, thedestination file system 120 can reorder the storage blocks 115 in theimage stream 230, similar to the “Tape Backup” embodiment describedherein.

Tape Backup. In a second preferred embodiment of volume copying (hereincalled “tape backup”), the destination file system 120 can include atape device or other similar long-term storage device. The system 100can copy storage blocks 115 from the mass storage 113 to that long-termstorage device, providing a backup copy of the file system 114 that canbe restored at a later time.

When performing tape backup, the system 100 creates an image stream 230,and copies the selected storage blocks 115 from the mass storage 113 atthe file server 110 to a sequence of new locations on the destinationfile system 120. Because the destination file system 120 includes one ormore tape drives, the system 100 creates and transmits a tableindicating which locations on the mass storage 113 correspond to whichother locations on the destination file system 120.

Similar to transfer of an image stream 230 using a network communicationprotocol, the destination file system 120 can add additional informationto the image stream 230 for recording on tape. This additionalinformation can include tape headers and tape gaps, blocking orclustering of storage blocks 115 for recording on tape, and reformattingof storage blocks 115 for recording on tape.

File Backup. In a third preferred embodiment of volume copying (hereincalled “file backup”), the image stream 230 can be copied to a new filewithin a file system 114, either at the file server 110 or at a filesystem 114 on the destination file system 120.

Similar to tape backup, the destination file system 120 can addadditional information to the image stream 230 for recording in an file.This additional information can include file metadata useful for thefile system 114 to locate storage blocks 115 within the file.

Volume Mirroring. The system 100 can be disposed to create image streams230 for copying the file system 114 to the destination file system 120coupled to a second file server 110 on a frequent basis, thus providinga mirror copy of the file system 114.

In a preferred embodiment, the mirror copy of the file system 114 can beused for takeover by a second file server 110 from the first file server110, such as for example if the first file server 110 fails.

When performing volume mirroring, the system 100 first transfers animage stream 230 representing a complete file system 114 from the fileserver 110 to the destination file system 120. The system 100 thenperiodically transfers image streams 230 representing incrementalchanges to that file system 114 from the file server 110 to thedestination file system 120. The destination file system 120 is able toreconstruct a most recent form of the consistent file system 114 fromthe initial full image stream 230 and the sequence of incremental imagestreams 230.

It is possible to perform volume mirroring using volume copying of afull storage image 230 and a sequence of incremental storage images 230.However, determining the storage blocks 115 to be included in anincremental storage images 230 can take substantial time for arelatively large file system 114, if done by logical subtraction.

As used herein, a “mark-on-allocate storage image” is a subset of asnapshot, the member storage blocks being those that have been added toa snapshot that originally formed a consistent file system.

In a preferred embodiment, rather than using logical subtraction, asdescribed above, at the time the incremental storage images 230 is aboutto be transferred, the file server 110 maintains a separate“mark-on-allocate” storage image 230. The mark-on-allocate storage image230 is constructed by setting a bit for each storage block 115, as it isadded to the consistent file system 114. The mark-on-allocate storageimage 230 does not need to be stored on the mass storage 113, includedin the block map, or otherwise backed-up; it can be reconstructed fromother storage images 230 already at the file server 110.

When an incremental storage image 230 is transferred, a firstmark-on-allocate storage image 230 is used to determine which storageblocks 115 to include in the storage image 230 for transfer. A secondmark-on-allocate storage image 230 is used to record changes to the filesystem 114 while the transfer is performed. After the transfer isperformed, the first and second mark-on-allocate storage images 230exchange roles.

Full Mirroring. In a first preferred embodiment of volume mirroring(herein called “full mirroring”), the destination file system 120includes a disk drive or other similar accessible storage device.

Upon the initial transfer of the full storage image 230 from the fileserver 110, the destination file system 120 creates a copy of theconsistent file system 114. Upon the sequential transfer of eachincremental storage image 230 from the file server 110, the destinationfile system 120 updates its copy of the consistent file system 114. Thedestination file system 120 thus maintains its copy of the file system114 nearly up to date, and can be inspected at any time.

When performing full mirroring, similar to disk copying, the system 100creates an image stream 230, and copies the selected storage blocks 115from the mass storage 113 at the file server 110 to correspondinglocations on the destination file system 120.

Incremental Mirroring. In a second preferred embodiment of volumemirroring (herein called “incremental mirroring”), the destination filesystem 120 can include both (1) a tape device or other relatively slowstorage device, and (2) a disk drive or other relatively fast storagedevice.

As used herein, an “incremental mirror” of a first file system is a basestorage image from the first file system, and at least one incrementalstorage image from the first file system, on two storage media ofsubstantially different types. Thus, a complete copy of the first filesystem can be reconstructed from the two or more objects.

Upon the initial transfer of the full storage image 230 from the fileserver 110, the destination file system 120 copies a complete set ofstorage blocks 115 from the mass storage 113 to that relatively slowstorage device. Upon the sequential transfer of each incremental storageimage 230 from the file server 110, the destination file system 120copies incremental sets of storage blocks 115 from the mass storage 113to the relatively fast storage device. Thus, the full set of storageblocks 115 plus the incremental sets of storage blocks 115 collectivelyrepresent an up-to-date file system 114 but do not require an entireduplicate disk drive.

When performing incremental mirroring, for the base storage image 230,the system 100 creates an image stream 230, and copies the selectedstorage blocks 115 from the mass storage 113 at the file server 110 to aset of new locations on the relatively slow storage device. The system100 writes the image stream 230, including storage block locationinformation, to the destination file system 120. In a preferredembodiment, the system 100 uses a tape as an intermediate destinationstorage medium, so that the base storage image 230 can be stored for asubstantial period of time without having to occupy disk space.

For each incremental storage image 230, the system 100 creates a newimage stream 230, and copies the selected storage blocks 115 from themass storage 113 at the file server 110 to a set of new locations on theaccessible storage device. Incremental storage images 230 are createdcontinuously and automatically at periodic times that are relativelyclose together.

The incremental storage images 230 are received at the destination filesystem 120, which unpacks them and records the copied storage blocks 115in an incremental mirror data structure. As each new incremental storageimage 230 is copied, copied storage blocks 115 overwrite the equivalentstorage blocks 115 from earlier incremental storage images 230. In apreferred embodiment, the incremental mirror data structure includes asparse file structure including only those storage blocks 115 that aredifferent from the base storage image 230.

In a preferred embodiment, the incremental storage images 230 aretransmitted to the destination file system 120 with a data structureindicating a set of storage blocks 115 that were deallocated (that is,removed) from the file system on the file server 110. Thus, the imagesare mark-on-deallocate images of the storage blocks. In response to thisdata structure, the destination file system 120 removes those indicatedstorage blocks 115 from its incremental mirror data structure. Thisallows the destination file system 120 to maintain the incrementalmirror data structure at a size no larger than approximately the actualdifferences between a current file system at the file server 110 and thebase storage image 230 from the file server 110.

Consistency Points. When performing either full mirroring or incrementalmirroring, it can occur that the transfer of a storage image 230 takeslonger than the time needed for the file server 110 to update itsconsistent file system 114 from a first consistency point to a secondconsistency point. Consistency points are described in further detail inthe WAFL Disclosures.

In a preferred embodiment, the file server 110 does not attempt tocreate a storage image 230 and to transfer storage blocks 115 for everyconsistency point. Instead, after a transfer of a storage image 230, thefile server 110 determines the most recent consistency point (oralternatively, determines the next consistency point) as the effectivenext consistency point. The file server 110 uses the effective nextconsistency point to determine any incremental storage image 230 for anext transfer.

Volume Replication. The destination file system 120 can include a diskdrive or other accessible storage device. The system 100 can copystorage blocks from the mass storage 113 to that accessible storagedevice at a signal from the destination file system 120, to providereplicated copies of the file system 114 for updated (read-only) use byother file servers 110.

The file server 110 maintains a set of selected master snapshots 210. Amaster snapshot 210 is a snapshot 210 whose existence can be known bythe destination file system 120, so that the destination file system 120can be updated with reference to the file system 114 maintained at thefile server 110. In a preferred embodiment, each master snapshot 210 isdesignated by an operator command at the file server 110, and isretained for a relatively long time, such as several months or a year.

In a preferred embodiment, at a minimum, each master snapshot 210 isretained until all known destination file systems 120 have been updatedpast that master snapshot 210. A master snapshot 210 can be designatedas a shadow snapshot 210, but in such cases destination file systems 120are taken off-line during update of the master shadow snapshot 210. Thatis, destination file systems 120 wait for completion of the update ofthat master shadow snapshot 210 before they are allowed to request anupdate from that master shadow snapshot 210.

The destination file system 120 generates a message (such as uponcommand of an operator or in response to initialization or self-test)that it transmits to the file server 110, requesting an update of thefile system 114. The message includes a newest master snapshot 210 towhich the destination file system 120 has most recently synchronized.The message can also indicate that there is no such newest mastersnapshot 210.

The file server 110 determines any incremental changes that haveoccurred to the file system 114 from the newest master snapshot 210 atthe destination file system 120 to the newest master snapshot 210 at thefile server 110. In response to this determination, the file server 110determines a storage image 230 including storage blocks 115 for transferto the destination file system 120, so as to update the copy of the filesystem 114 at the destination file system 120.

If there is no such newest master snapshot 210, the system 100 performsvolume copying for a full copy of the file system 114 represented by thenewest master snapshot 210 at the file server 110. Similarly, if theoldest master snapshot 210 at the file server 110 is newer than thenewest master snapshot 210 at the destination file system 120, thesystem 100 performs volume copying for a full copy of the file system114.

After volume replication, the destination file system 120 updates itsmost recent master snapshot 210 to be the most recent master snapshot210 from the file server 110.

Volume replication is well suited to uploading upgrades to a publiclyaccessible database, document, or web site. Those destination filesystems 120, such as mirror sites, can then obtain the uploaded upgradesperiodically, when they are initialized, or upon operator command at thedestination file system 120. If the destination file systems 120 are notin communication with the file server 110 for a substantial period oftime, when communication is re-established, the destination file systems120 can perform volume replication with the file server 110 to obtain asubstantially up-to-date copy of the file system 114.

In a first preferred embodiment of volume replication (herein called“simple replication”), the destination file system 120 communicatesdirectly (using a direct communication link, a LAN, a WAN, or acombination thereof) with the file server 110.

In a second preferred embodiment of volume replication (herein called“multiple replication”), a first destination file system communicatesdirectly (using a direct communication link, a LAN, a WAN, or acombination thereof) with a second destination file system. The seconddestination file system acts like the file server 110 to perform simplereplication for the first destination file system.

A sequence of such destination file systems ultimately terminates in adestination file system that communicates directly with the file server110 and performs simple replication. The sequence of destination filesystems thus forms a replication hierarchy, such as in a directed graphor a tree of file severs 110.

In alternative embodiments, the system 100 can also perform one or morecombinations of these techniques.

In a preferred embodiment, the file server 110 can maintain a set ofpointers to snapshots 210, naming those snapshots 210 and having theproperty that references to the pointers are functionally equivalent toreferences to the snapshots 210 themselves. For example, one of thepointers can have a name such as “master,” so that the newest mastersnapshot 210 at the file server 110 can be changed simultaneously forall destination file systems. Thus, all destination file systems cansynchronize to the same master snapshot 210.

Shadow Snapshots

The system 100 includes the possibility of designating selectedsnapshots 210 as “shadow” snapshots 210.

As used herein, a “shadow snapshot” is a subset of a snapshot, themember storage blocks no longer forming a consistent file system. Thus,at one time the member storage blocks of the snapshot did form aconsistent file system, but at least some of the member storage blockshave been removed from that snapshot.

A shadow snapshot 210 has the property that the file server 110 canreuse the storage blocks 115 in the snapshot 210 whenever needed. Ashadow snapshot 210 can be used as the base of an incremental storageimage 230. In such cases, storage blocks 115 might have been removedfrom the shadow snapshot 210 due to reuse by the file system 110. Itthus might occur that the incremental storage image 230 resulting fromlogically subtraction using the shadow snapshot 210 includes storageblocks 115 that are not strictly necessary (having been removed from theshadow snapshot 210 they are not subtracted out). However, all storageblocks 115 necessary for the incremental storage image 230 will still beincluded.

For regular snapshots 210, the file server 110 does not reuse thestorage blocks 115 in the snapshot 210 until the snapshot 210 isreleased. Even if the storage blocks 115 in the snapshot 210 are nolonger part of the active file system, the file server retains themwithout change. Until released, each regular snapshot 210 preserves aconsistent file system 114 that can be accessed at a later time.

However, for shadow snapshots 210, the file server 110 can reuse thestorage blocks 115 in the shadow snapshot 210. When one of those storageblocks 115 is reused, the file server 110 clears the bit in the shadowsnapshot 210 for that storage block 115. Thus, each shadow snapshot 210represents a set of storage blocks 115 from a consistent file system 114that have not been changed in the active file system 114 since theshadow snapshot 210 was made. Because storage blocks 115 can be reused,the shadow snapshot 210 does not retain the property of representing aconsistent file system 114. However, because the file server 110 canreuse those storage blocks 115, the shadow snapshot 210 does not causeany storage blocks 115 on the mass storage 113 to be permanentlyoccupied.

Method of Operation

FIG. 3 shows a process flow diagram of a method for file system imagetransfer.

A method 300 is performed is performed by the file server 110 and thedestination file system 120, and includes a set of flow points andprocess steps as described herein.

Generality of Operational Technique

In each of the file system image transfer techniques, the method 300performs three operations:

Select a storage image 220, in response to a first file system (or asnapshot thereof) to have an operation performed thereon.

Form an image stream 230 in response to the storage image 220. Performan operation on the image stream 230, such as backup or restore withinthe first file system, or copying or transfer to a second file system.

Reconstruct the first file system (or the snapshot thereof) in responseto the image stream 230.

As shown herein, each of these steps is quite general in itsapplication.

In the first (selection) step, the storage image 220 selected can be acomplete file system or can be a subset thereof. The subset can be anincrement to the complete file system, such as those storage blocks thathave been changed, or can be another type of subset. The storage image220 can be selected a single time, such as for a backup operation, orrepeatedly, such as for a mirroring operation. The storage image 220 canbe selected in response to a process at a sending file server or at areceiving file server.

For example, as shown herein, the storage image 220 selected can be fora full backup or copying of an entire file system, or can be forincremental backup or incremental mirroring of a file system. Thestorage image 220 selected can be determined by a sending file server,or can be determined in response to a request by a receiving file server(or set of receiving file servers).

In the second (operational) step, the image stream 230 can be selectedso as to optimize the operation. The image stream 230 can be selectedand ordered to optimize transfer to different types of media, tooptimize transfer rate, or to optimize reliability. In a preferredembodiment, the image stream 230 is optimized to maximize transfer ratefrom parallel disks in a RAID disk system.

In the third (reconstruction) step, the image stream 230 can bereconstructed into a complete file system, or can be reconstructed intoan increment of a file system. The reconstruction step can be performedimmediately or after a delay, can be performed in response to theprocess that initiated the selection step, or can be performedindependently in response to other needs.

Selecting A Storage Image

In each of the file system image transfer techniques, the method 300selects a storage image 220 to be transferred.

At a flow point 301, the file server 110 is ready to select a storageimage 220 for transfer.

At a step 302, the file server 110 forms a logical sum LS of a set ofstorage images 220 A1+A2, thus LS=A1+A2. The logical sum LS can alsoinclude any plurality of storage images 220, such as A1+A2+A3+A4, thusfor example LS=A1+A2+A3+A4.

At a step 303, the file server 110 determines if the transfer is a fulltransfer or an incremental transfer. If the transfer is incremental, themethod 300 continues with the next step. If the transfer is a fulltransfer, the method 300 continues with the flow point 380.

At a step 304, the file server 110 forms a logical difference LD of thelogical sum LS and a base storage image 220 B, thus LD=LS−B. The basestorage image 220 B comprises a snapshot 210.

At a flow point 305, the file server 110 has selected a storage image230 for transfer.

Volume Copying

At a flow point 310, the file server 110 is ready to perform a volumecopying operation.

At a step 311, the file server 111 selects a storage image 220 fortransfer, as described with regard to the flow point 370 through theflow point 380. If the volume copying operation is a full volume copy,the storage image 220 selected is for a full transfer. If the volumecopying operation is an incremental volume copy, the storage image 220selected is for an incremental transfer.

At a step 312, the file server 110 determines if the volume is to becopied to disk or to tape.

If the volume is to be copied to disk, the method 300 continues with thestep 313.

If the volume is to be copied to tape, the method 300 continues with thestep 314.

At a step 313, the file server 110 creates an image stream 230 for theselected storage image 220. In a preferred embodiment, the storageblocks 115 in the image stream 230 are ordered for transfer to disk.Each storage block 115 is associated with a VBN (virtual block number)for identification. The method 300 continues with the step 315.

At a step 314, the file server 110 performs the same functions as in thestep 313, except that the storage blocks 115 in the image stream 230 areordered for transfer to tape.

At a step 315, the file server 110 copies the image stream 230 to thedestination file system 120 (disk or tape).

If the image stream 230 is copied to disk, the file server 110preferably places each storage block 115 in an equivalent position onthe target disk(s) as it was on the source disk(s), similar to whatwould happen on retrieval from tape.

In a preferred embodiment, the file server 110 copies the image stream230 to the destination file system 120 using a communication protocolknown to both the file server 110 and the destination file system 120,such as TCP. As noted herein, the image stream 230 used with thecommunication protocol is similar to the image stream 230 used for tapebackup, but can include additional messages or packets foracknowledgement or retransmission of data.

The destination file system 120 presents the image stream 230 directlyto a restore element, which copies the image stream 230 onto thedestination file system 120 target disk(s) as they were on the sourcedisk(s). Because a consistent file system 114 is copied from the fileserver 110 to the destination file system 120, the storage blocks 115 inthe image stream 230 can be used directly as a consistent file system114 when they arrive at the destination file system 120.

The destination file system 120 might have to alter some inter-blockpointers, responsive to the VBN of each storage block 115, if some orall of the target storage blocks 115 are recorded in different physicallocations on disk from the source storage blocks 115.

If the image stream 230 is copied to tape, the file server 110preferably places each storage block 115 in a position on the targettape so that it can be retrieved by its VBN. When the storage blocks 115are eventually retrieved from tape into a disk file server 110, they arepreferably placed in equivalent positions on the target disk(s) as theywere on the source disk(s).

The destination file system 120 records the image stream 230 directlyonto tape, along with a set of block number information for each storageblock 115. The destination file system 120 can later retrieve selectedstorage blocks 115 from tape and place them onto a disk file server 110.Because a consistent file system 114 is copied from the file server 110to the destination file system 120, the storage blocks 115 in the imagestream 230 can be restored directly to disk when later retrieved fromtape at the destination file system 120.

The destination file system 120 might have to alter some inter-blockpointers, responsive to the VBN of each storage block 115, if some orall of the target storage blocks 115 are retrieved from tape andrecorded in different physical locations on disk from the source storageblocks 115. The destination file system 120 recorded this information inheader data that it records onto tape.

At a flow point 320, the file server 110 has completed the volumecopying operation.

Volume Mirroring

At a flow point 330, the file server 110 is ready to perform a volumemirroring operation.

At a step 331, the file server 110 performs a full volume copyingoperation, as described with regard to the flow point 310 through theflow point 320. The volume copying operation is performed for a fullcopy of the file system 114.

If the function to be performed is full mirroring, the file server 110performs the full volume copying operation to disk as the targetdestination file system 120.

If the function to be performed is incremental mirroring, the fileserver 110 performs the full volume copying operation to tape as thetarget destination file system 120.

At a step 332, the file server 110 sets a mirroring timer forincremental update for the volume mirroring operation.

At a step 333, the mirroring timer is hit, and the file server 110begins the incremental update for the volume mirroring operation.

At a step 334, the file server 110 performs an incremental volumecopying operation, as described with regard to the flow point 310through the flow point 320. The volume copying operation is performedfor an incremental upgrade of the file system 114.

The incremental volume copying operation is performed with disk as thetarget destination file system 120.

If the initial full volume copying operation was performed to disk, thedestination file system 120 increments its copy of the file system 114to include the incremental storage image 220.

If the initial full volume copying operation was performed to tape, thedestination file system 120 records the incremental storage image 220and integrates it into an incremental mirror data structure, asdescribed above, for possibly later incrementing its copy of the filesystem 114.

At a step 335, the file server 110 copies the image stream 230 to thetarget destination file system 120. The method 300 returns to the step332, at which step the file server 110 resets the mirroring timer, andthe method 300 continues.

When the destination file system 120 receives the image stream 230, itrecords the storage blocks 115 in that image stream 230 similar to theprocess of volume copying, as described with regard to the step 315.

If the method 300 is halted (by an operator command or otherwise), themethod 300 completes at the flow point 340.

At a flow point 340, the file server 110 has completed the volumemirroring operation.

Reintegration of Incremental Mirror

At a flow point 370, the file server 110 is ready to restore a filesystem from the base storage image 220 and the incremental mirror datastructure.

At a step 371, the file server 110 reads the base storage image 220 intoits file system.

At a step 372, the file server 110 reads the incremental mirror datastructure into its file system and uses that data structure to updatethe base storage image 220.

At a step 373, the file server 110 remounts the file system that wasupdated using the incremental mirror data structure.

At a flow point 380, the file server 110 is ready to continue operationswith the file system restored from the base storage image 220 and theincremental mirror data structure.

Volume Replication

At a flow point 350, the file server 110 is ready to perform a volumereplication operation.

At a step 351, the destination file system 120 initiates the volumereplication operation. The destination file system 120 sends anindicator of its newest master snapshot 210 to the file server 110, andrequests the file server 110 to perform the volume replicationoperation.

At a step 352, the file server 110 determines if it needs to perform avolume replication operation to synchronize with a second file server110. In this case, the second file server 110 takes the role of thedestination file system 120, and initiates the volume replicationoperation with regard to the first file server 110.

At a step 353, the file server 110 determines its newest master snapshot210, and its master snapshot 210 corresponding to the master snapshot210 indicated by the destination file system 120.

If the file server 110 has at least one master snapshot 210 older thanthe master snapshot 210 indicated by the destination file system 120, itselects the corresponding master snapshot 210 as the newest one ofthose.

In this case, the method proceeds with the step 354.

If the file server 110 does not have at least one master snapshot 210older than the master snapshot 210 indicated by the destination filesystem 120 (or if the destination file system 120 did not indicate anymaster snapshot 210), it does not select any master snapshot 210 as acorresponding master snapshot.

In this case, the method proceeds with the step 355.

At a step 354, the file server 110 performs an incremental volumecopying operation, responsive to the incremental difference between theselected corresponding master snapshot 210, and the newest mastersnapshot 210 it has available. The method 300 proceeds with the flowpoint 360.

At a step 355, the file server 110 performs a full volume copyingoperation, responsive to the newest master snapshot 210 it hasavailable. The method 300 proceeds with the flow point 360.

At a flow point 360, the file server 110 has completed the volumereplication operation. The destination file system 120 updates itsmaster snapshot 210 to correspond to the master snapshot 210 that wasused to make the file system transfer from the file server 110.

Technical Appendix

A technical appendix, tided “WAFL Image Transfer,” and having theinventors named as authors, forms a part of this specification, and ishereby incorporated by reference as if fully set forth herein.

Alternative Embodiments

Although preferred embodiments are disclosed herein, many variations arepossible which remain within the concept, scope, and spirit of theinvention, and these variations would become clear to those skilled inthe art after perusal of this application.

What is claimed is:
 1. A method for identifying storage blocks in a filesystem having a plurality of storage blocks, comprising the steps of:identifying a first storage image indicating a first set of memberstorage blocks selected from the plurality, the first storage imageincluding a first bit plane indicating which member storage blocks arein the first storage image; identifying a second storage imageindicating a second set of member storage blocks selected from theplurality, the second storage image including a second bit planeindicating which member storage blocks are in the second storage image;performing a logical operation on the first bit plane and the second bitplane to determine a logical difference between the first storage imageand the second storage image; wherein at least the first storage imageor the second storage image indicates member storage blocks forming aconsistent file system other than an active file system.
 2. A method foridentifying storage blocks in a file system having a plurality ofstorage blocks, comprising the steps of: identifying a first storageimage indicating a first set of member storage blocks selected from theplurality, the first storage image including a first bit planeindicating which member storage blocks are in the first storage image;identifying a second storage image indicating a second set of memberstorage blocks selected from the plurality, the second storage imageincluding a second bit plane indicating which member storage blocks arein the second storage image; performing a logical operation on the firstbit plane and the second bit plane to determine a logical sum of thefirst storage image and the second storage image; wherein at least thefirst storage image or the second storage image indicates member storageblocks forming a consistent file system other than an active filesystem.
 3. A memory storing information including instructions, theinstructions executable by a processor to identify storage blocks in afile system having a plurality of storage blocks, the instructionscomprising: identifying a first storage image indicating a first set ofmember storage blocks selected from the plurality, the first storageimage including a first bit plane indicating which member storage blocksare in the first storage image; identifying a second storage imageindicating a second set of member storage blocks selected from theplurality, the second storage image including a second bit planeindicating which member storage blocks are in the second storage image;performing a logical operation on the first bit plane and the second bitplane to determine a logical difference between the first storage imageand the second storage image; wherein at least the first storage imageor the second storage image indicates member storage blocks forming aconsistent file system other than an active file system.
 4. A memorystoring information including instructions, the instructions executableby a processor to identify storage blocks in a file system having aplurality of storage blocks, the instructions comprising: identifying afirst storage image indicating a first set of member storage blocksselected from the plurality, the first storage image including a firstbit plane indicating which member storage blocks are in the firststorage image; identifying a second storage image indicating a secondset of member storage blocks selected from the plurality, the secondstorage image including a second bit plane indicating which memberstorage blocks are in the second storage image; performing a logicaloperation on the first bit plane and the second bit plane to determine alogical sum of the first storage image and the second storage image;wherein at least the first storage image or the second storage imageindicates member storage blocks forming a consistent file system otherthan an active file system.
 5. An apparatus including: a storage mediumthat stores a file system having a plurality of storage blocks; aprocessor that executes instructions; and a memory that stores theinstructions, the instructions executable by the processor to identifystorage blocks in the file system, the instructions comprising: (a)identifying a first storage image indicating a first set of memberstorage blocks selected from the plurality, the first storage imageincluding a first bit plane indicating which member storage blocks arein the first storage image, (b) identifying a second storage imageindicating a second set of member storage blocks selected from theplurality, the second storage image including a second bit planeindicating which member storage blocks are in the second storage image,(c) performing a logical operation on the first bit plane and the secondbit plane to determine a logical difference between the first storageimage and the second storage image, wherein at least the first storageimage or the second storage image indicates member storage blocksforming a consistent file system other than an active file system.
 6. Anapparatus including: a storage medium that stores a file system having aplurality of storage blocks; a processor that executes instructions; anda memory that stores the instructions, the instructions executable bythe processor to identify storage blocks in the file system, theinstructions comprising: (a) identifying a first storage imageindicating a first set of member storage blocks selected from theplurality, the first storage image including a first bit planeindicating which member storage blocks are in the first blocks selectedfrom the plurality, the second storage image including a second bitplane indicating which member storage blocks are in the second storageimage, (c) performing a logical operation on the first bit plane and thesecond bit plane to determine a logical sum of the first storage imageand the second storage image, wherein at least the first storage imageor the second storage image indicates member storage blocks forming aconsistent file system other than an active file system.
 7. A method asin claim 1, wherein the first bit plane and the second bit plane eachform a column of a collection of bit planes, each row in the collectionrepresenting a storage block.
 8. A method as in claim 2, wherein thefirst bit plane and the second bit plane each form a column of acollection of bit planes, each row in the collection representing astorage block.
 9. A memory as in claim 3, wherein the first bit planeand the second bit plane each form a column of a collection of bitplanes, each row in the collection representing a storage block.
 10. Amemory as in claim 4, wherein the first bit plane and the second bitplane each form a column of a collection of bit planes, each row in thecollection representing a storage block.
 11. An apparatus as in claim 5,wherein the first bit plane and the second bit plane each form a columnof a collection of bit planes, each row in the collection representing astorage block.
 12. An apparatus as in claim 6, wherein the first bitplane and the second bit plane each form a column of a collection of bitplanes, each row in the collection representing a storage block.